Prometheus 集成 Nginx 监控

背景

在之前文章：Nginx 日志分析之 Loki 中提到过如何收集并分析 Nginx 日志，并与 Grafana 进行了集成。

日志和监控是可观测性的不同方向，监控获取的信息不如日志详实，但比日志更轻量、收集速度更快。

下文介绍如何使用 nginx-module-vts 对 Nginx 进行监控。

‍

环境

操作系统：CentOS 7

Nginx 版本：1.27.3

nginx-module-vts 版本：0.2.3

IP：192.168.2.10

‍

实践

Nginx 编译和配置修改

Nginx 的指标采集依赖第三方模块 nginx-module-vts，需要下载后编译到 Nginx 中。

Nginx 源码包：github.com/nginx/nginx/releases/download/release-1.27.3/nginx-1….

nginx-module-vts下载链接：github.com/vozlt/nginx-module-vts/releases

‍

由于之前已经编译安装过 nginx用于添加 Loki 进行 IP 归属分析，查看 nginx 当前的编译参数：

# nginx -V
nginx version: nginx/1.27.3
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) 
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-compat --with-debug --with-file-aio --with-google_perftools_module --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E' --with-http_geoip_module --add-module=/data/nginx/nginx-1.27.3/nginx-dav-ext-module 

‍

编译安装：

cd /data/nginx/nginx-1.27.3
wget https://github.com/vozlt/nginx-module-vts/archive/refs/tags/v0.2.3.tar.gz
tar xf v0.2.3.tar.gz

# 在之前基础上新增编译参数 --add-module=/data/nginx/nginx-1.27.3/nginx-module-vts-0.2.3
./configure  --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-compat --with-debug --with-file-aio --with-google_perftools_module --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E' --with-http_geoip_module --add-module=/data/nginx/nginx-1.27.3/nginx-dav-ext-module --add-module=/data/nginx/nginx-1.27.3/nginx-module-vts-0.2.3
make && make install 

# 确认新的二进制编译参数
# nginx -V
nginx version: nginx/1.27.3
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) 
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-compat --with-debug --with-file-aio --with-google_perftools_module --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E' --with-http_geoip_module --add-module=/data/nginx/nginx-1.27.3/nginx-dav-ext-module --add-module=/data/nginx/nginx-1.27.3/nginx-module-vts-0.2.3

‍

修改 nginx 配置文件以开启流量统计：

# http块 中添加配置
    # 启用vts
    vhost_traffic_status_zone;
    vhost_traffic_status_filter_by_host on;   #开启此功能，会根据不同的server_name进行流量的统计，否则默认会把流量全部计算到第一个上。
    vhost_traffic_status_filter on;
    vhost_traffic_status_filter_by_set_key $status $server_name;

# location中添加配置以暴露收集的数据，这里使用了一个新的子配置文件
# cat /etc/nginx/conf.d/vts.conf 
server {
    listen 192.168.2.10:80; 

    location /status {
        vhost_traffic_status_display;
        vhost_traffic_status_display_format html;
    }
}

# 重启 nginx
systemctl restart nginx

‍

VTS 介绍

vts v0.2.0 及其之后的版本，已经添加了对 prometheus 的指标输出，不需要再使用 nginx-vts-exporter 将 vts 暴露的数据格式化后交由 prometheus 采集。

它提供对虚拟主机状态信息的访问。包含当前状态，例如服务器、上游、缓存。这类似于nginx plus的实时活动监控。内置的html也取自旧版本的演示页面。

‍

支持以如下方式访问：

• /status/format/json
  • If you request /status/format/json​, will respond with a JSON document containing the current activity data for using in live dashboards and third-party monitoring tools.
• /status/format/html
  • If you request /status/format/html​, will respond with the built-in live dashboard in HTML that requests internally to /status/format/json​.
• /status/format/jsonp
  • If you request /status/format/jsonp​, will respond with a JSONP callback function containing the current activity data for using in live dashboards and third-party monitoring tools.
• /status/format/prometheus
  • If you request /status/format/prometheus​, will respond with a prometheus document containing the current activity data.
• /status/control
  • If you request /status/control​, will respond with a JSON document after it reset or delete zones through a query string. See the Control.

‍

‍

一般情况下，使用 /status/format/prometheus​ 收集指标，web 浏览器访问 /status​ 已经足够。

​​

‍

​​

‍

在生产环境中，以域名为维度比以 upstream 更清晰，所以上文在 nginx 中使用了如下三行配置：

• vhost_traffic_status_filter_by_host on;
• vhost_traffic_status_filter on;
• vhost_traffic_status_filter_by_set_key $status$server_name;

‍

Promrtheus 配置

添加新job：

# vim /etc/prometheus/prometheus.yml
- job_name: 'nginx-module-vts'
  static_configs:
    - targets: ['192.168.2.10']
  metrics_path: /status/format/prometheus
  scheme: http

# 重载prometheus
systemctl reload prometheus

‍

Grafana

我自己绘制了一副图表，效果如下：

​​

有需要可自取 json 并导入：open.opshub.cn/wordpress/grafana-nginx.json

‍

引用链接

• Nginx 日志分析之 Loki
• nginx-module-vts
• github.com/nginx/nginx/releases/download/release-1.27.3/nginx-1....
• github.com/vozlt/nginx-module-vts/releases
• prometheus
• Control
• open.opshub.cn/wordpress/grafana-nginx.json